Learning in a Large Function Space: Privacy-Preserving Mechanisms for SVM Learning

Several recent studies in privacy-preserving learning have considered the trade-o between utility or risk and the level of di erential privacy guaranteed by mechanisms for statistical query processing. In this paper we study this trade-o in private Support Vector Machine (SVM) learning. We present two e cient mechanisms, one for the case of nite-dimensional feature mappings and one for potentially in nite-dimensional feature mappings with translation-invariant kernels. For the case of translation-invariant kernels, the proposed mechanism minimizes regularized empirical risk in a random Reproducing Kernel Hilbert Space whose kernel uniformly approximates the desired kernel with high probability. This technique, borrowed from large-scale learning, allows the mechanism to respond with a nite encoding of the classi er, even when the function class is of in nite VC dimension. Di erential privacy is established using a proof technique from algorithmic stability. Utility the mechanism's response function is pointwise -close to non-private SVM with probability 1− δ is proven by appealing to the smoothness of regularized empirical risk minimization with respect to small perturbations to the feature mapping. We conclude with a lower bound on the optimal di erential privacy of the SVM. This negative result states that for any δ, no mechanism can be simultaneously ( , δ)-useful and β-di erentially private for small and small β.